Skip macroblock coding

ABSTRACT

Various techniques and tools for encoding and decoding (e.g., in a video encoder/decoder) binary information (e.g., skipped macroblock information) are described. In some embodiments, the binary information is arranged in a bit plane, and the bit plane is coded at the picture/frame layer. The encoder and decoder process the binary information and, in some embodiments, switch coding modes. For example, the encoder and decoder use normal, row-skip, column-skip, or differential modes, or other and/or additional modes. In some embodiments, the encoder and decoder define a skipped macroblock as a predicted macroblock whose motion is equal to its causally predicted motion and which has zero residual error. In some embodiments, the encoder and decoder use a raw coding mode to allow for low-latency applications.

RELATED APPLICATION INFORMATION

This application is a continuation of U.S. patent application Ser. No.15/342,790, filed Nov. 3, 2016, which is a continuation of Ser. No.14/746,594, filed Jun. 22, 2015, now U.S. Pat. No. 9,538,189, which is acontinuation of U.S. patent application Ser. No. 14/292,540, filed May30, 2014, now U.S. Pat. No. 9,088,785, which is a continuation of U.S.patent application Ser. No. 13/866,918, filed Apr. 19, 2013, now U.S.Pat. No. 8,781,240, which is a continuation of U.S. patent applicationSer. No. 12/495,481, filed Jun. 30, 2009, now U.S. Pat. No. 8,428,374,which is a continuation of U.S. patent application Ser. No. 11/495,354,filed Jul. 27, 2006, now U.S. Pat. No. 7,555,167, which is a divisionalof U.S. patent application Ser. No. 10/321,415, filed Dec. 16, 2002, nowU.S. Pat. No. 7,200,275, the disclosure of which is incorporated byreference. U.S. patent application Ser. No. 10/321,415 claims thebenefit of U.S. Provisional Patent Application Ser. No. 60/341,674,entitled “Techniques and Tools for Video Encoding and Decoding,” filedDec. 17, 2001, the disclosure of which is incorporated by reference.U.S. patent application Ser. No. 10/321,415 also claims the benefit ofU.S. Provisional Patent Application Ser. No. 60/377,712, entitled “SkipMacroblock Coding,” filed May 3, 2002, the disclosure of which isincorporated by reference.

TECHNICAL FIELD

Techniques and tools for encoding/decoding binary information in videocoding/decoding applications are described. For example, a video encoderencodes skipped macroblock information.

BACKGROUND

Digital video consumes large amounts of storage and transmissioncapacity. A typical raw digital video sequence includes 15 or 30 framesper second. Each frame can include tens or hundreds of thousands ofpixels (also called pels). Each pixel represents a tiny element of thepicture. In raw form, a computer commonly represents a pixel with 24bits. Thus, the number of bits per second, or bit rate, of a typical rawdigital video sequence can be 5 million bits/second or more.

Most computers and computer networks lack the resources to process rawdigital video. For this reason, engineers use compression (also calledcoding or encoding) to reduce the bit rate of digital video. Compressioncan be lossless, in which quality of the video does not suffer butdecreases in bit rate are limited by the complexity of the video. Or,compression can be lossy, in which quality of the video suffers butdecreases in bit rate are more dramatic. Decompression reversescompression.

In general, video compression techniques include intraframe compressionand interframe compression. Intraframe compression techniques compressindividual frames, typically called I-frames, or key frames. Interframecompression techniques compress frames with reference to precedingand/or following frames, and are called typically called predictedframes, P-frames, or B-frames.

Microsoft Corporation's Windows Media Video, Version 7 [“WMV7”] includesa video encoder and a video decoder. The WMV7 encoder uses intraframeand interframe compression, and the WMV7 decoder uses intraframe andinterframe decompression.

A. Intraframe Compression in WMV7

FIG. 1 illustrates block-based intraframe compression (100) of a block(105) of pixels in a key frame in the WMV7 encoder. A block is a set ofpixels, for example, an 8×8 arrangement of pixels. The WMV7 encodersplits a key video frame into 8×8 blocks of pixels and applies an 8×8Discrete Cosine Transform [“DCT”] (110) to individual blocks such as theblock (105). A DCT is a type of frequency transform that converts the8×8 block of pixels (spatial information) into an 8×8 block of DCTcoefficients (115), which are frequency information. The DCT operationitself is lossless or nearly lossless. Compared to the original pixelvalues, however, the DCT coefficients are more efficient for the encoderto compress since most of the significant information is concentrated inlow frequency coefficients (conventionally, the upper left of the block(115)) and many of the high frequency coefficients (conventionally, thelower right of the block (115)) have values of zero or close to zero.

The encoder then quantizes (120) the DCT coefficients, resulting in an8×8 block of quantized DCT coefficients (125). For example, the encoderapplies a uniform, scalar quantization step size to each coefficient,which is analogous to dividing each coefficient by the same value androunding. For example, if a DCT coefficient value is 163 and the stepsize is 10, the quantized DCT coefficient value is 16. Quantization islossy. The reconstructed DCT coefficient value will be 160, not 163.Since low frequency DCT coefficients tend to have higher values,quantization results in loss of precision but not complete loss of theinformation for the coefficients. On the other hand, since highfrequency DCT coefficients tend to have values of zero or close to zero,quantization of the high frequency coefficients typically results incontiguous regions of zero values. In addition, in some cases highfrequency DCT coefficients are quantized more coarsely than lowfrequency DCT coefficients, resulting in greater loss ofprecision/information for the high frequency DCT coefficients.

The encoder then prepares the 8×8 block of quantized DCT coefficients(125) for entropy encoding, which is a form of lossless compression. Theexact type of entropy encoding can vary depending on whether acoefficient is a DC coefficient (lowest frequency), an AC coefficient(other frequencies) in the top row or left column, or another ACcoefficient.

The encoder encodes the DC coefficient (126) as a differential from theDC coefficient (136) of a neighboring 8×8 block, which is a previouslyencoded neighbor (e.g., top or left) of the block being encoded. (FIG. 1shows a neighbor block (135) that is situated to the left of the blockbeing encoded in the frame.) The encoder entropy encodes (140) thedifferential.

The entropy encoder can encode the left column or top row of ACcoefficients as a differential from a corresponding column or row of theneighboring 8×8 block. FIG. 1 shows the left column (127) of ACcoefficients encoded as a differential (147) from the left column (137)of the neighboring (to the left) block (135). The differential codingincreases the chance that the differential coefficients have zerovalues. The remaining AC coefficients are from the block (125) ofquantized DCT coefficients. The encoder scans (150) the 8×8 block (145)of predicted, quantized AC DCT coefficients into a one-dimensional array(155) and then entropy encodes the scanned AC coefficients using avariation of run length coding (160). The encoder selects an entropycode from one or more run/level/last tables (165) and outputs theentropy code.

A key frame contributes much more to bit rate than a predicted frame. Inlow or mid-bit rate applications, key frames are often criticalbottlenecks for performance, so efficient compression of key frames iscritical.

FIG. 2 illustrates a disadvantage of intraframe compression such asshown in FIG. 1. In particular, exploitation of redundancy betweenblocks of the key frame is limited to prediction of a subset offrequency coefficients (e.g., the DC coefficient and the left column (ortop row) of AC coefficients) from the left (220) or top (230)neighboring block of a block (210). The DC coefficient represents theaverage of the block, the left column of AC coefficients represents theaverages of the rows of a block, and the top row represents the averagesof the columns. In effect, prediction of DC and AC coefficients as inWMV7 limits extrapolation to the row-wise (or column-wise) averagesignals of the left (or top) neighboring block. For a particular row(221) in the left block (220), the AC coefficients in the left DCTcoefficient column for the left block (220) are used to predict theentire corresponding row (211) of the block (210).

B. Interframe Compression in WMV7

Interframe compression in the WMV7 encoder uses block-based motioncompensated prediction coding followed by transform coding of theresidual error. FIGS. 3 and 4 illustrate the block-based interframecompression for a predicted frame in the WMV7 encoder. In particular,FIG. 3 illustrates motion estimation for a predicted frame (310) andFIG. 4 illustrates compression of a prediction residual for amotion-estimated block of a predicted frame.

The WMV7 encoder splits a predicted frame into 8×8 blocks of pixels.Groups of 4 8×8 blocks form macroblocks. For each macroblock, a motionestimation process is performed. The motion estimation approximates themotion of the macroblock of pixels relative to a reference frame, forexample, a previously coded, preceding frame. In FIG. 3, the WMV7encoder computes a motion vector for a macroblock (315) in the predictedframe (310). To compute the motion vector, the encoder searches in asearch area (335) of a reference frame (330). Within the search area(335), the encoder compares the macroblock (315) from the predictedframe (310) to various candidate macroblocks in order to find acandidate macroblock that is a good match. The encoder can checkcandidate macroblocks every pixel or every ½ pixel in the search area(335), depending on the desired motion estimation resolution for theencoder. Other video encoders check at other increments, for example,every ¼ pixel. For a candidate macroblock, the encoder checks thedifference between the macroblock (315) of the predicted frame (310) andthe candidate macroblock and the cost of encoding the motion vector forthat macroblock. After the encoder finds a good matching macroblock, theblock matching process ends. The encoder outputs the motion vector(entropy coded) for the matching macroblock so the decoder can find thematching macroblock during decoding. When decoding the predicted frame(310), a decoder uses the motion vector to compute a predictionmacroblock for the macroblock (315) using information from the referenceframe (330). The prediction for the macroblock (315) is rarely perfect,so the encoder usually encodes 8×8 blocks of pixel differences (alsocalled the error or residual blocks) between the prediction macroblockand the macroblock (315) itself.

FIG. 4 illustrates the computation and encoding of an error block (435)for a motion-estimated block in the WMV7 encoder. The error block (435)is the difference between the predicted block (415) and the originalcurrent block (425). The encoder applies a DCT (440) to error block(435), resulting in 8×8 block (445) of coefficients. Even more than wasthe case with DCT coefficients for pixel values, the significantinformation for the error block (435) is concentrated in low frequencycoefficients (conventionally, the upper left of the block (445)) andmany of the high frequency coefficients have values of zero or close tozero (conventionally, the lower right of the block (445)).

The encoder then quantizes (450) the DCT coefficients, resulting in an8×8 block of quantized DCT coefficients (455). The quantization stepsize is adjustable. Again, since low frequency DCT coefficients tend tohave higher values, quantization results in loss of precision, but notcomplete loss of the information for the coefficients. On the otherhand, since high frequency DCT coefficients tend to have values of zeroor close to zero, quantization of the high frequency coefficientsresults in contiguous regions of zero values. In addition, in some caseshigh frequency DCT coefficients are quantized more coarsely than lowfrequency DCT coefficients, resulting in greater loss ofprecision/information for the high frequency DCT coefficients.

The encoder then prepares the 8×8 block (455) of quantized DCTcoefficients for entropy encoding. The encoder scans (460) the 8×8 block(455) into a one dimensional array (465) with 64 elements, such thatcoefficients are generally ordered from lowest frequency to highestfrequency, which typical creates long runs of zero values.

The encoder entropy encodes the scanned coefficients using a variationof run length coding (470). The encoder selects an entropy code from oneor more run/level/last tables (475) and outputs the entropy code.

When the motion vector for a macroblock is zero (i.e., no motion) and noresidual block information is transmitted for the macroblock, theencoder uses a 1-bit skip macroblock flag for the macroblock. For manykinds of video content (e.g., low motion and/or low bitrate video), thisreduces bitrate by avoiding the transmission of motion vector andresidual block information. The encoder puts the skip macroblock flagfor a macroblock at the macroblock layer in the output bitstream, alongwith other information for the macroblock.

FIG. 5 shows the decoding process (500) for an inter-coded block. Due tothe quantization of the DCT coefficients, the reconstructed block (575)is not identical to the corresponding original block. The compression islossy.

In summary of FIG. 5, a decoder decodes (510, 520) entropy-codedinformation representing a prediction residual using variable lengthdecoding and one or more run/level/last tables (515). The decoderinverse scans (530) a one-dimensional array (525) storing theentropy-decoded information into a two-dimensional block (535). Thedecoder inverse quantizes and inverse discrete cosine transforms(together, 540) the data, resulting in a reconstructed error block(545). In a separate path, the decoder computes a predicted block (565)using motion vector information (555) for displacement from a referenceframe. The decoder combines (570) the predicted block (555) with thereconstructed error block (545) to form the reconstructed block (575).

When the decoder receives a skip macroblock flag for a macroblock, thedecoder skips computing a prediction and decoding residual blockinformation for the macroblock. Instead, the decoder uses correspondingpixel data from the location of the macroblock in the reference frame.

The amount of change between the original and reconstructed frame istermed the distortion and the number of bits required to code the frameis termed the rate. The amount of distortion is roughly inverselyproportional to the rate. In other words, coding a frame with fewer bits(greater compression) will result in greater distortion and vice versa.One of the goals of a video compression scheme is to try to improve therate-distortion—in other words to try to achieve the same distortionusing fewer bits (or the same bits and lower distortion).

Although the use of skip macroblock flags in WMV7 typically reducesbitrate for many kinds of video content, it is less than optimal in somecircumstances. In many cases, it fails to exploit redundancy in skipmacroblock flags from macroblock to macroblock, for example, whenskipped macroblocks occur in bunches in a picture. Also, WMV7 ignoresmotion prediction for macroblocks in predicted frames when themacroblocks are skipped, which hurts the efficiency of compression ofpredicted frames in some cases.

C. Standards for Video Compression and Decompression

Aside from WMV7, several international standards relate to videocompression and decompression. These standards include the MotionPicture Experts Group [“MPEG”] 1, 2, and 4 standards and the H.261,H.262, and H.263 standards from the International TelecommunicationUnion [“ITU”]. Like WMV7, these standards use a combination ofintraframe and interframe compression, although the standards typicallydiffer from WMV7 in the details of the compression techniques used.

Some international standards recognize skipping coding of macroblocks asa tool to be used in video compression and decompression. For additionaldetail about skip macroblock coding in the standards, see the standards'specifications themselves.

The skipped macroblock coding in the above standards typically reducesbitrate for many kinds of video content, but is less than optimal insome circumstances. In many cases, it fails to exploit redundancy inskip macroblock flags from macroblock to macroblock, for example, whenskipped macroblocks occur in bunches in a picture. Also, it ignoresmotion prediction for macroblocks in predicted macroblocks/pictures whenthe macroblocks are skipped, which hurts the efficiency of compressionof predicted macroblocks/pictures in some cases.

Given the critical importance of video compression and decompression todigital video, it is not surprising that video compression anddecompression are richly developed fields. Whatever the benefits ofprevious video compression and decompression techniques, however, theydo not have the advantages of the following techniques and tools.

SUMMARY

In summary, the detailed description is directed to various techniquesand tools for encoding and decoding (e.g., in a video encoder/decoder)binary information. The binary information may comprise bits indicatingwhether a video encoder or decoder skips certain macroblocks in a videoframe. Or, the binary information may comprise bits indicating motionvector resolution for macroblocks (e.g. 1-MV or 4-MV), interlace mode(e.g., field or frame), or some other information. Binary informationmay be encoded on a frame-by-frame basis or on some other basis.

In some embodiments, the binary information is arranged in a bit plane.For example, the bit plane is coded at the picture/frame layer.Alternatively, the binary information is arranged in some other wayand/or coded at a different layer. The encoder and decoder process thebinary information. The binary information may comprise macroblock-levelinformation. Alternatively, the encoder and decoder process bit planesof block-level, sub-block-level, or pixel-level information.

In some embodiments, the encoder and decoder switch coding modes. Forexample, the encoder and decoder use normal, row-skip, or column-skipmode. The different modes allow the encoder and decoder to exploitredundancy in the binary information. Alternatively, the encoder anddecoder use other and/or additional modes such as differential modes. Toincrease efficiency, the encoder and decoder may use a bit planeinversion technique in some modes.

In some embodiments, the encoder and decoder define a skipped macroblockas a predicted macroblock whose motion is equal to its causallypredicted motion and which has zero residual error. Alternatively, theencoder and decoder define a skipped macroblock as a predictedmacroblock with zero motion and zero residual error.

In some embodiments, the encoder and decoder use a raw coding mode toallow for low-latency applications. For example, in the raw coding mode,encoded macroblocks can be transmitted to the decoder right away,without having to wait until all macroblocks in the frame/picture areencoded. The encoder and decoder can switch between the raw coding modeand other modes.

The various techniques and tools can be used in combination orindependently. In particular, the application describes twoimplementations of skipped macroblock encoding and decoding, along withcorresponding bitstream syntaxes. Different embodiments implement one ormore of the described techniques and tools.

Additional features and advantages will be made apparent from thefollowing detailed description of different embodiments that proceedswith reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing block-based intraframe compression of an 8×8block of pixels according to prior art.

FIG. 2 is a diagram showing prediction of frequency coefficientsaccording to the prior art.

FIG. 3 is a diagram showing motion estimation in a video encoderaccording to the prior art.

FIG. 4 is a diagram showing block-based interframe compression for an8×8 block of prediction residuals in a video encoder according to theprior art.

FIG. 5 is a diagram showing block-based intraframe decompression for an8×8 block of prediction residuals according to the prior art.

FIG. 6 is a block diagram of a suitable computing environment in whichseveral described embodiments may be implemented.

FIG. 7 is a block diagram of a generalized video encoder system used inseveral described embodiments.

FIG. 8 is a block diagram of a generalized video decoder system used inseveral described embodiments.

FIG. 9 is a chart showing the bitstream elements that make up the Ppicture layer according to the first implementation.

FIG. 10 is a flowchart showing a technique for encoding skippedmacroblock information in a video encoder having plural skip-macroblockcoding modes.

FIG. 11 is a flowchart showing a technique for decoding skippedmacroblock information encoded by a video encoder having pluralskip-macroblock coding modes.

FIG. 12 shows an example of a skipped macroblock coding frame.

FIG. 13 is a flowchart showing a technique for encoding in normalskip-macroblock coding mode.

FIG. 14 is a flowchart showing a technique for encoding in arow-prediction skip-macroblock coding mode.

FIG. 15 is a code listing showing pseudo-code for row-predictiondecoding of skipped macroblock information.

FIG. 16 is a flowchart showing a technique for encoding in acolumn-prediction skip-macroblock coding mode.

FIG. 17 is a code listing showing pseudo-code for column-predictiondecoding of skipped macroblock information.

FIG. 18 is a flowchart showing a technique for determining whether toskip coding of certain macroblocks in a video encoder.

FIG. 19 is a flowchart showing a technique for encoding binaryinformation in a bit plane in a row-skip coding mode.

FIG. 20 is a flowchart showing a technique for encoding binaryinformation in a bit plane in a column-skip coding mode.

FIG. 21 is a flowchart showing a technique for encoding binaryinformation in a bit plane in a normal-2 coding mode.

FIGS. 22, 23 and 24 show examples of frames of binary information tiledin normal-6 mode.

FIG. 25 is a flowchart showing a technique for encoding binaryinformation in a bit plane in a normal-6 coding mode.

FIG. 26 is a flowchart showing a technique for encoding binaryinformation in a differential coding mode.

FIG. 27 is a flowchart showing a technique for decoding binaryinformation encoded in a differential coding mode.

FIG. 28 is a flowchart showing a technique for selectively encodingbinary information in raw coding mode for low latency applications.

DETAILED DESCRIPTION

Described embodiments relate to techniques and tools for encoding anddecoding (e.g., in a video encoder/decoder) binary information. Thebinary information may comprise bits indicating whether a video encoderor decoder skips certain macroblocks in a video frame. Or, the binaryinformation may comprise bits indicating motion vector resolution formacroblocks (e.g. 1-MV or 4-MV), interlace mode (e.g., field or frame),or some other information. Binary information may be encoded on aframe-by-frame basis or on some other basis.

In some embodiments, the binary information is arranged in a bit plane.The bit plane is coded at the picture/frame layer. Alternatively, thebinary information is arranged in some other way and/or coded at adifferent layer.

In some embodiments, the encoder and decoder switch coding modes. Forexample, the encoder and decoder use normal, row-skip, or column-skipmodes. The different modes allow the encoder and decoder to exploitredundancy in the binary information. Alternatively, the encoder anddecoder use other and/or additional modes.

In some embodiments, the encoder and decoder define a skipped macroblockas a predicted macroblock whose motion is equal to its causallypredicted motion and which has zero residual error. Alternatively, theencoder and decoder define a skipped macroblock as a predictedmacroblock with zero motion and zero residual error.

In some embodiments, instead of efficient frame/picture-level coding, araw coding mode is permitted to allow for low-latency applications. Inthe raw coding mode, encoded macroblocks can be transmitted to thedecoder right away, without having to wait until all macroblocks in theframe/picture are encoded.

In some embodiments, the encoder and decoder process bit planes ofmacroblock level information. Alternatively, the encoder and decoderprocess bit planes of block, sub-block, or pixel-level information.

The various techniques and tools can be used in combination orindependently. In particular, the application describes twoimplementations of skipped macroblock encoding and decoding, along withcorresponding bitstream syntaxes. Different embodiments implement one ormore of the described techniques and tools.

In described embodiments, the video encoder and decoder perform varioustechniques. Although the operations for these techniques are typicallydescribed in a particular, sequential order for the sake ofpresentation, it should be understood that this manner of descriptionencompasses minor rearrangements in the order of operations, unless aparticular ordering is required. For example, operations describedsequentially may in some cases be rearranged or performed concurrently.Moreover, for the sake of simplicity, flowcharts typically do not showthe various ways in which particular techniques can be used inconjunction with other techniques.

In described embodiments, the video encoder and decoder use variousflags and signals in a bitstream. While specific flags and signals aredescribed, it should be understood that this manner of descriptionencompasses different conventions (e.g., 0's rather than 1's) for theflags and signals.

I. Computing Environment

FIG. 6 illustrates a generalized example of a suitable computingenvironment (600) in which several of the described embodiments may beimplemented. The computing environment (600) is not intended to suggestany limitation as to scope of use or functionality, as the techniquesand tools may be implemented in diverse general-purpose orspecial-purpose computing environments.

With reference to FIG. 6, the computing environment (600) includes atleast one processing unit (610) and memory (620). In FIG. 6, this mostbasic configuration (630) is included within a dashed line. Theprocessing unit (610) executes computer-executable instructions and maybe a real or a virtual processor. In a multi-processing system, multipleprocessing units execute computer-executable instructions to increaseprocessing power. The memory (620) may be volatile memory (e.g.,registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flashmemory, etc.), or some combination of the two. The memory (620) storessoftware (680) implementing an encoder or decoder, such as a videoencoder or decoder.

A computing environment may have additional features. For example, thecomputing environment (600) includes storage (640), one or more inputdevices (650), one or more output devices (660), and one or morecommunication connections (670). An interconnection mechanism (notshown) such as a bus, controller, or network interconnects thecomponents of the computing environment (600). Typically, operatingsystem software (not shown) provides an operating environment for othersoftware executing in the computing environment (600), and coordinatesactivities of the components of the computing environment (600).

The storage (640) may be removable or non-removable, and includesmagnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any othermedium which can be used to store information and which can be accessedwithin the computing environment (600). The storage (640) storesinstructions for the software (680) implementing the encoder or decoder.

The input device(s) (650) may be a touch input device such as akeyboard, mouse, pen, or trackball, a voice input device, a scanningdevice, or another device that provides input to the computingenvironment (600). For audio or video encoding, the input device(s)(650) may be a sound card, video card, TV tuner card, or similar devicethat accepts audio or video input in analog or digital form, or a CD-ROMor CD-RW that reads audio or video samples into the computingenvironment (600). The output device(s) (660) may be a display, printer,speaker, CD-writer, or another device that provides output from thecomputing environment (600).

The communication connection(s) (670) enable communication over acommunication medium to another computing entity. The communicationmedium conveys information such as computer-executable instructions,audio or video input or output, or other data in a modulated datasignal. A modulated data signal is a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia include wired or wireless techniques implemented with anelectrical, optical, RF, infrared, acoustic, or other carrier.

The techniques and tools can be described in the general context ofcomputer-readable media. Computer-readable media are any available mediathat can be accessed within a computing environment. By way of example,and not limitation, with the computing environment (600),computer-readable media include memory (620), storage (640),communication media, and combinations of any of the above.

The techniques and tools can be described in the general context ofcomputer-executable instructions, such as those included in programmodules, being executed in a computing environment on a target real orvirtual processor. Generally, program modules include routines,programs, libraries, objects, classes, components, data structures, etc.that perform particular tasks or implement particular abstract datatypes. The functionality of the program modules may be combined or splitbetween program modules as desired in various embodiments.Computer-executable instructions for program modules may be executedwithin a local or distributed computing environment.

For the sake of presentation, the detailed description uses terms like“determine,” “select,” “reconstruct,” and “inform” to describe computeroperations in a computing environment. These terms are high-levelabstractions for operations performed by a computer, and should not beconfused with acts performed by a human being. The actual computeroperations corresponding to these terms vary depending onimplementation.

II. Generalized Video Encoder and Decoder

FIG. 7 is a block diagram of a generalized video encoder (700) and FIG.8 is a block diagram of a generalized video decoder (800).

The relationships shown between modules within the encoder and decoderindicate the main flow of information in the encoder and decoder; otherrelationships are not shown for the sake of simplicity. In particular,FIGS. 7 and 8 usually do not show side information indicating theencoder settings, modes, tables, etc. used for a video sequence, frame,macroblock, block, etc. Such side information is sent in the outputbitstream, typically after entropy encoding of the side information. Theformat of the output bitstream can be Windows Media Video version 8format or another format.

The encoder (700) and decoder (800) are block-based and use a 4:2:0macroblock format with each macroblock including 4 luminance 8×8luminance blocks (at times treated as one 16×16 macroblock) and two 8×8chrominance blocks. Alternatively, the encoder (700) and decoder (800)are object-based, use a different macroblock or block format, or performoperations on sets of pixels of different size or configuration than 8×8blocks and 16×16 macroblocks.

Depending on implementation and the type of compression desired, modulesof the encoder or decoder can be added, omitted, split into multiplemodules, combined with other modules, and/or replaced with like modules.In alternative embodiments, encoder or decoders with different modulesand/or other configurations of modules perform one or more of thedescribed techniques.

A. Video Encoder

FIG. 7 is a block diagram of a general video encoder system (700). Theencoder system (700) receives a sequence of video frames including acurrent frame (705), and produces compressed video information (795) asoutput. Particular embodiments of video encoders typically use avariation or supplemented version of the generalized encoder (700).

The encoder system (700) compresses predicted frames and key frames. Forthe sake of presentation, FIG. 7 shows a path for key frames through theencoder system (700) and a path for forward-predicted frames. Many ofthe components of the encoder system (700) are used for compressing bothkey frames and predicted frames. The exact operations performed by thosecomponents can vary depending on the type of information beingcompressed.

A predicted frame [also called p-frame, b-frame for bi-directionalprediction, or inter-coded frame] is represented in terms of prediction(or difference) from one or more other frames. A prediction residual isthe difference between what was predicted and the original frame. Incontrast, a key frame [also called i-frame, intra-coded frame] iscompressed without reference to other frames.

If the current frame (705) is a forward-predicted frame, a motionestimator (710) estimates motion of macroblocks or other sets of pixelsof the current frame (705) with respect to a reference frame, which isthe reconstructed previous frame (725) buffered in the frame store(720). In alternative embodiments, the reference frame is a later frameor the current frame is bi-directionally predicted. The motion estimator(710) can estimate motion by pixel, ½ pixel, ¼ pixel, or otherincrements, and can switch the resolution of the motion estimation on aframe-by-frame basis or other basis. The resolution of the motionestimation can be the same or different horizontally and vertically. Themotion estimator (710) outputs as side information motion information(715) such as motion vectors. A motion compensator (730) applies themotion information (715) to the reconstructed previous frame (725) toform a motion-compensated current frame (735). The prediction is rarelyperfect, however, and the difference between the motion-compensatedcurrent frame (735) and the original current frame (705) is theprediction residual (745). Alternatively, a motion estimator and motioncompensator apply another type of motion estimation/compensation.

A frequency transformer (760) converts the spatial domain videoinformation into frequency domain (i.e., spectral) data. For block-basedvideo frames, the frequency transformer (760) applies a discrete cosinetransform [“DCT”] or variant of DCT to blocks of the pixel data orprediction residual data, producing blocks of DCT coefficients.Alternatively, the frequency transformer (760) applies anotherconventional frequency transform such as a Fourier transform or useswavelet or subband analysis. In embodiments in which the encoder usesspatial extrapolation (not shown in FIG. 7) to encode blocks of keyframes, the frequency transformer (760) can apply a re-orientedfrequency transform such as a skewed DCT to blocks of predictionresiduals for the key frame. In other embodiments, the frequencytransformer (760) applies an 8×8, 8×4, 4×8, or other size frequencytransforms (e.g., DCT) to prediction residuals for predicted frames.

A quantizer (770) then quantizes the blocks of spectral datacoefficients. The quantizer applies uniform, scalar quantization to thespectral data with a step-size that varies on a frame-by-frame basis orother basis. Alternatively, the quantizer applies another type ofquantization to the spectral data coefficients, for example, anon-uniform, vector, or non-adaptive quantization, or directly quantizesspatial domain data in an encoder system that does not use frequencytransformations. In addition to adaptive quantization, the encoder (700)can use frame dropping, adaptive filtering, or other techniques for ratecontrol.

If a given macroblock in a predicted frame has no information of certaintypes (e.g., no motion information for the macroblock and no residualinformation), the encoder (700) may encode the macroblock as a skippedmacroblock. If so, the encoder signals the skipped macroblock in theoutput bitstream of compressed video information (795).

When a reconstructed current frame is needed for subsequent motionestimation/compensation, an inverse quantizer (776) performs inversequantization on the quantized spectral data coefficients. An inversefrequency transformer (766) then performs the inverse of the operationsof the frequency transformer (760), producing a reconstructed predictionresidual (for a predicted frame) or a reconstructed key frame. If thecurrent frame (705) was a key frame, the reconstructed key frame istaken as the reconstructed current frame (not shown). If the currentframe (705) was a predicted frame, the reconstructed prediction residualis added to the motion-compensated current frame (735) to form thereconstructed current frame. The frame store (720) buffers thereconstructed current frame for use in predicting the next frame. Insome embodiments, the encoder applies a deblocking filter to thereconstructed frame to adaptively smooth discontinuities in the blocksof the frame.

The entropy coder (780) compresses the output of the quantizer (770) aswell as certain side information (e.g., motion information (715),spatial extrapolation modes, quantization step size). Typical entropycoding techniques include arithmetic coding, differential coding,Huffman coding, run length coding, LZ coding, dictionary coding, andcombinations of the above. The entropy coder (780) typically usesdifferent coding techniques for different kinds of information (e.g., DCcoefficients, AC coefficients, different kinds of side information), andcan choose from among multiple code tables within a particular codingtechnique.

The entropy coder (780) puts compressed video information (795) in thebuffer (790). A buffer level indicator is fed back to bit rate adaptivemodules.

The compressed video information (795) is depleted from the buffer (790)at a constant or relatively constant bit rate and stored for subsequentstreaming at that bit rate. Therefore, the level of the buffer (790) isprimarily a function of the entropy of the filtered, quantized videoinformation, which affects the efficiency of the entropy coding.Alternatively, the encoder system (700) streams compressed videoinformation immediately following compression, and the level of thebuffer (790) also depends on the rate at which information is depletedfrom the buffer (790) for transmission.

Before or after the buffer (790), the compressed video information (795)can be channel coded for transmission over the network. The channelcoding can apply error detection and correction data to the compressedvideo information (795).

B. Video Decoder

FIG. 8 is a block diagram of a general video decoder system (800). Thedecoder system (800) receives information (895) for a compressedsequence of video frames and produces output including a reconstructedframe (805). Particular embodiments of video decoders typically use avariation or supplemented version of the generalized decoder (800).

The decoder system (800) decompresses predicted frames and key frames.For the sake of presentation, FIG. 8 shows a path for key frames throughthe decoder system (800) and a path for forward-predicted frames. Manyof the components of the decoder system (800) are used for compressingboth key frames and predicted frames. The exact operations performed bythose components can vary depending on the type of information beingcompressed.

A buffer (890) receives the information (895) for the compressed videosequence and makes the received information available to the entropydecoder (880). The buffer (890) typically receives the information at arate that is fairly constant over time, and includes a jitter buffer tosmooth short-term variations in bandwidth or transmission. The buffer(890) can include a playback buffer and other buffers as well.Alternatively, the buffer (890) receives information at a varying rate.Before or after the buffer (890), the compressed video information canbe channel decoded and processed for error detection and correction.

The entropy decoder (880) entropy decodes entropy-coded quantized dataas well as entropy-coded side information (e.g., motion information(815), spatial extrapolation modes, quantization step size), typicallyapplying the inverse of the entropy encoding performed in the encoder.Entropy decoding techniques include arithmetic decoding, differentialdecoding, Huffman decoding, run length decoding, LZ decoding, dictionarydecoding, and combinations of the above. The entropy decoder (880)frequently uses different decoding techniques for different kinds ofinformation (e.g., DC coefficients, AC coefficients, different kinds ofside information), and can choose from among multiple code tables withina particular decoding technique.

If the frame (805) to be reconstructed is a forward-predicted frame, amotion compensator (830) applies motion information (815) to a referenceframe (825) to form a prediction (835) of the frame (805) beingreconstructed. For example, the motion compensator (830) uses amacroblock motion vector to find a macroblock in the reference frame(825). A frame buffer (820) stores previous reconstructed frames for useas reference frames. The motion compensator (830) can compensate formotion at pixel, ½ pixel, ¼ pixel, or other increments, and can switchthe resolution of the motion compensation on a frame-by-frame basis orother basis. The resolution of the motion compensation can be the sameor different horizontally and vertically. Alternatively, a motioncompensator applies another type of motion compensation. The predictionby the motion compensator is rarely perfect, so the decoder (800) alsoreconstructs prediction residuals.

When the decoder needs a reconstructed frame for subsequent motioncompensation, the frame store (820) buffers the reconstructed frame foruse in predicting the next frame. In some embodiments, the encoderapplies a deblocking filter to the reconstructed frame to adaptivelysmooth discontinuities in the blocks of the frame.

An inverse quantizer (870) inverse quantizes entropy-decoded data. Ingeneral, the inverse quantizer applies uniform, scalar inversequantization to the entropy-decoded data with a step-size that varies ona frame-by-frame basis or other basis. Alternatively, the inversequantizer applies another type of inverse quantization to the data, forexample, a non-uniform, vector, or non-adaptive quantization, ordirectly inverse quantizes spatial domain data in a decoder system thatdoes not use inverse frequency transformations.

An inverse frequency transformer (860) converts the quantized, frequencydomain data into spatial domain video information. For block-based videoframes, the inverse frequency transformer (860) applies an inverse DCT[“IDCT”] or variant of IDCT to blocks of the DCT coefficients, producingpixel data or prediction residual data for key frames or predictedframes, respectively. Alternatively, the frequency transformer (860)applies another conventional inverse frequency transform such as aFourier transform or uses wavelet or subband synthesis. In embodimentsin which the decoder uses spatial extrapolation (not shown in FIG. 8) todecode blocks of key frames, the inverse frequency transformer (860) canapply a re-oriented inverse frequency transform such as a skewed IDCT toblocks of prediction residuals for the key frame. In other embodiments,the inverse frequency transformer (860) applies an 8×8, 8×4, 4×8, orother size inverse frequency transforms (e.g., IDCT) to predictionresiduals for predicted frames.

When a skipped macroblock is signaled in the bitstream of information(895) for a compressed sequence of video frames, the decoder (800)reconstructs the skipped macroblock without using the information (e.g.,motion information and/or residual information) normally included in thebitstream for non-skipped macroblocks.

III. First Implementation

In a first implementation, a video encoder and decoder encode anddecode, respectively, skipped macroblock information with improvedefficiency. The skipped macroblock information is signaled at thepicture layer in the video bitstream, which allows the encoder toexploit redundancy in the skipped macroblock information. Also, theencoder and decoder select between multiple coding modes for encodingand decoding the skipped macroblock information.

A. Picture Layer Coding of Skipped Macroblock Information

In the first implementation, a compressed video sequence is made up ofdata structured into four hierarchical layers. From top to bottom thelayers are: 1) sequence layer; 2) picture layer; 3) macroblock layer;and 4) block layer. At the picture layer, data for each picture consistsof a picture header followed by data for the macroblock layer.(Similarly, at the macroblock layer, data for each macroblock consistsof a macroblock header followed by the block layer.) While some of thebitstream elements for I pictures and P pictures are identical, othersappear only in P pictures, and vice versa.

FIG. 9 shows the bitstream elements that make up the P-picture layer(900). Table 1 briefly describes the bitstream elements of the P-picturelayer (900).

TABLE 1 Bitstream elements of the P-picture layer in firstimplementation Field Description PTYPE (910) Picture type PQUANT (912)Picture quantizer scale SMBC (920) Skipped macroblock code SMB (930)Skipped macroblock field CPBTAB (940) Coded block pattern table MVRES(942) Motion vector resolution TTMBF (944) Macroblock-level transformtype flag TTFRM (946) Frame-level transform type DCTACMBF (948)Macroblock-level DCT AC coding set flag DCTACFRM (950) Frame-level DCTAC coding set index DCTDCTAB (952) Intra DCT DC table MVTAB (954) Motionvector table MB LAYER (960) Macroblock layer

In particular, the P-picture layer (900) includes a Skipped Macroblockfield (“SMB”) (930) for the macroblocks in the P picture as well as aSkipped Macroblock Code (“SMBC”) field (920) that signals the codingmode for the skipped macroblock field (930). The SMBC field (920) ispresent only in P-picture headers. SMBC (920) is a 2-bit value thatsignals one of four modes used for indicating the skipped macroblocks inthe frame. In the first implementation, the fixed length codes (“FLCs”)for the skipped macroblock coding modes are as follows:

TABLE 2 Skipped macroblock coding mode code table in firstimplementation SMBC FLC Skipped Bit Coding Mode 00 No skipped bit coding01 Normal skipped bit coding 10 Row-prediction (or, “row-skip”) skippedbit coding 11 Column-prediction (or, “column-skip”) skipped bit coding

If the coding mode is normal, row-prediction, or column-prediction, thenthe next field in the bitstream is the SMB field (930) containing theskipped macroblock information. So, the SMB field is present only inP-picture headers and only if SMBC signals normal, row-prediction, orcolumn-prediction skipped macroblock coding. If SMBC signals normalcoding, then the size of the SMB field is equal to the number ofmacroblocks in the frame. If SMBC signals row-prediction orcolumn-prediction, then the size of the SMB is variable as describedbelow.

The skipped macroblock information informs the decoder as to whichmacroblocks in the frame are not present in the macroblock layer. Forthese macroblocks, the decoder will copy the corresponding macroblockpixel data from the reference frame when reconstructing that macroblock.

B. Switching Coding Modes for Skipped Macroblock Information

As described above, the SMBC field (920) signals the coding mode for theskipped macroblock field (930). More generally, FIG. 10 shows atechnique (1000) for encoding skipped macroblock information in a videoencoder having multiple skip-macroblock coding modes. FIG. 11 shows acorresponding technique (1100) for decoding skipped macroblockinformation encoded by a video encoder having plural skip-macroblockcoding modes.

With reference to FIG. 10, the encoder selects a skip-macroblock codingmode for coding skipped macroblock information (1010). For example, inthe first implementation, the skipped macroblock coding modes include amode where no macroblocks are skipped, a normal mode, a row-prediction(or, “row-skip”) mode, and a column-prediction (or “column-skip”) mode.After the coding mode is selected, the encoder encodes the skippedmacroblock information (1020). The encoder selects coding modes on apicture-by-picture basis. Alternatively, the encoder selects codingmodes on some other basis (e.g., at the sequence level). When theencoder is done encoding the skipped macroblock information (1030),encoding ends.

With reference to FIG. 11, the decoder determines the skip-macroblockcoding mode used by the encoder to encode the skipped macroblockinformation (1110). The decoder then decodes the skipped macroblockinformation (1120). The decoder determines coding modes on apicture-by-picture basis. Alternatively, the decoder determines codingmodes on some other basis (e.g., at the sequence level). When thedecoder is done decoding the skipped macroblock information (1130),decoding ends.

C. Coding Modes

In the first implementation, the skipped macroblock coding modes includea mode where no macroblocks are skipped, a normal mode, a row-prediction(or, “row-skip”) mode, and a column-prediction (or “column-skip”) mode.The following sections describe how skipped macroblock information isencoded in each mode with reference to FIG. 12, which shows an example(1200) of a skipped macroblock coding frame.

1. Normal Skipped Macroblock Coding Mode

In normal mode, the skipped/not-skipped status of each macroblock isrepresented with a bit. Therefore, the size of the SMB field in bits isequal to the number of macroblocks in the frame. The bit position withinthe SMB field corresponds to the raster scan order of the macroblockswithin the frame starting with the upper-left macroblock. A bit value of0 indicates that the corresponding macroblock is not skipped; a bitvalue of 1 indicates that the corresponding macroblock is skipped.

FIG. 13 shows a technique (1300) for encoding in normal skip-macroblockcoding mode. First, the encoder checks whether coding of a macroblockwill be skipped (1310). If so, the encoder adds a bit value of 1 to theSMB field to indicate that the corresponding macroblock is skipped(1320). Otherwise, the encoder adds a bit value of 0 to the SMB field toindicate that the corresponding macroblock is not skipped (1330). Whenthe encoder is done adding bits to the SMB field (1340), skip macroblockcoding ends.

As an example, using normal mode coding, the SMB field for the exampleframe (1200) in FIG. 12 would be encoded as: 010010111111111111010010.

2. Row-Prediction Skipped Macroblock Coding Mode

In row-prediction mode, the status of each macroblock row (from top tobottom) is indicated with a bit. If the bit is 1, then the row containsall skipped macroblocks and the status for the next row follows. If thebit equals 0, then the skipped/not skipped status for each macroblock inthat row is signaled with a bit. Therefore, a bit field equal in lengthto the number of macroblocks in a row follows. The bits in the bit fieldrepresent the macroblocks in left-to-right order. Again, a value of 0indicates that the corresponding macroblock is not skipped; a value of 1indicates that the corresponding macroblock is skipped.

FIG. 14 shows a technique (1400) for encoding in row-prediction (or,“row-skip”) macroblock coding mode. First, the encoder checks if a rowcontains all skipped macroblocks (1410). If so, the encoder adds anindicator bit of 1 to the SMB field (1420) and the status for the nextrow follows. If the row does not contain all skipped macroblocks, theencoder adds an indicator bit of 0 to the SMB field, and theskipped/not-skipped status for each macroblock in that row is signaledwith a bit (1430). When the encoder is done with all the rows in theframe (1440), the row-prediction encoding ends.

As for decoding, FIG. 15 shows pseudo-code (1500) illustratingrow-prediction decoding of the skipped macroblock information. In thepseudo-code (1500), the function get_bits(n) reads n bits from thebitstream and returns the value.

As an example, using row-prediction mode coding, the SMB field for theexample frame (1200) in FIG. 12 would be encoded as: 0010010110010010.

3. Column-Prediction Skipped Macroblock Coding Mode

In column-prediction mode, the status of each macroblock column (fromleft to right) is indicated with a bit. If the bit is 1, then the columncontains all skipped macroblocks and the status for the next columnfollows. If the bit equals 0, then the skipped/not skipped status foreach macroblock in that column is signaled with a bit. Therefore, a bitfield equal in length to the number of macroblocks in a column follows.The bits in the bit field represent the macroblocks in top-to-bottomorder. Again, a value of 0 indicates that the corresponding macroblockis not skipped; a value of 1 indicates that the corresponding macroblockis skipped.

FIG. 16 shows a technique (1600) for encoding in column-prediction (or,“column-skip”) macroblock coding mode. First, the encoder checks if thecolumn contains all skipped macroblocks (1610). If so, the encoder addsan indicator bit of 1 to the SMB field (1620) and the status for thenext column follows. If the column does not contain all skippedmacroblocks, the encoder adds an indicator bit of 0 to the SMB field,and the skipped/not-skipped status for each macroblock in that column issignaled with a bit (1630). When the encoder is done with all thecolumns in the frame (1640), the column-prediction encoding ends.

As for decoding, FIG. 17 shows pseudo-code (1700) illustratingcolumn-prediction decoding of the skipped macroblock information.

As an example, using column-prediction mode coding, the SMB field forthe example frame (1200) in FIG. 12 would be encoded as:0011010011000110100110.

IV. Second Implementation

In a second implementation, a video encoder and decoder encode anddecode, respectively, skipped macroblock information and/or other 2-Dbinary data with improved efficiency. The encoder and decoder define askipped macroblock as having a default motion (not necessarily zeromotion), which allows the encoder and decoder to skip more macroblocksin many cases. Efficient frame-level coding of bit planes indicatesskipped macroblock information and/or other 2-D binary data. Also, theencoder and decoder may use a raw (MB-level) coding option of skippedmacroblocks for low-latency applications.

A. Skip Bit Definition (Definition of Skipped Macroblock)

The second implementation includes a new definition of the concept of askipped macroblock. “Skip” refers to a condition in a bitstream where nofurther information needs to be transmitted at that level ofgranularity. A skipped macroblock (block) is a macroblock (block) thathas a default type, default motion, and default residual error. (Incomparison, in other implementations and standards, skipped macroblocksare predicted macroblocks with zero motion and zero residuals.

The new definition of skipped macroblock is a predicted macroblock whosemotion is equal to its causally predicted motion, and which has zeroresidual error. (The point of difference from the other definition isthe default motion is equal to the motion predictor, and this may notnecessarily be zero.)

For example, in some embodiments, predicted motion vectors for a currentmacroblock are taken from the macroblock directly above or directly tothe left of the current macroblock. Or, horizontal and verticalcomponents of the predictor are generated from the horizontal andvertical component-wise medians of the macroblocks the left, top, andtop right of the current macroblock.

The motion vectors of a skipped macroblock with four motion vectors(4MV) are given by their predictions performed sequentially in thenatural scan order. As with the one motion vector (1MV) case, the errorresiduals are zero.

FIG. 18 shows a technique (1800) for determining whether to skip codingof particular macroblocks in a video encoder according to the newdefinition of skipped macroblocks. First, the encoder checks whether thecurrent frame is an I-frame or a P-frame (1810). If the current frame isan I-frame, no macroblocks in the current frame are skipped (1820), andskip-macroblock coding for the frame ends.

On the other hand, if the current frame is a P-frame, the encoder checksfor macroblocks in the current frame that can be skipped. For a givenmacroblock, the encoder checks whether the motion vector for themacroblock is equal to the causally predicted motion vector for themacroblock (e.g., whether the differential motion vector for themacroblock is equal to zero) (1830). If the motion for a macroblock doesnot equal the causally predicted motion, the encoder does not skip themacroblock (1840). Otherwise, the encoder checks whether there is anyresidual to be encoded for the macroblock (1850). If there is a residualto be coded, the encoder does not skip the macroblock (1860). If thereis no residual for the macroblock, however, the encoder skips themacroblock (1870). The encoder continues to encode or skip macroblocksuntil encoding is done (1880).

B. Bit Plane Coding

In the second implementation, certain macroblock-specific information(including signaling skipped macroblocks) can be encoded in one bit permacroblock. The status for all macroblocks in a frame can be codedtogether as a bit plane and transmitted in the frame header.

In the second implementation, the encoder uses bit plane coding in threecases to signal information about macroblocks in a frame. The threecases are: 1) signaling skipped macroblocks, 2) signaling field or framemacroblock mode, and 3) signaling 1-MV or 4-MV motion vector mode foreach macroblock. This section describes bit plane coding for any of thethree cases and corresponding decoding.

Frame-level bit plane coding is used to encode two-dimensional binaryarrays. The size of each array is rowMB×colMB, where rowMB and colMB arethe number of macroblock rows and columns, respectively. Within thebitstream, each array is coded as a set of consecutive bits. One ofseven modes is used to encode each array, as enumerated in Table 3 anddescribed below.

TABLE 3 Coding modes in second implementation Coding Mode DescriptionRaw Coded as one bit per symbol Normal-2 Two symbols coded jointlyDiff-2 Differential coding of bit plane, followed by coding two residualsymbols jointly Normal-6 Six symbols coded jointly Diff-6 Differentialcoding of bit plane, followed by coding six residual symbols jointlyRow-skip One bit skip to signal rows with no set bits. Column-skip Onebit skip to signal columns with no set bits.

In the second implementation, the encoder uses three syntax elements toembed the information in a bit plane: MODE, INVERT and DATABITS.

The MODE field is a variable length code (“VLC”) that encodes the codingmode for the bit plane. For example, the VLC in the MODE fieldrepresents any of the seven coding modes enumerated in Table 3. To savebits, the encoder can assign shorter codes to more probable coding modesand longer codes to less probable coding modes. As noted above, the MODEfield is transmitted in the frame header.

The encoder and decoder switch between coding modes on a frame-by-framebasis. For example, the encoder and decoder switch between coding modesas like the encoder and decoder of the first implementation switchbetween skipped macroblock coding modes in FIGS. 10 and 11,respectively. Alternatively, the encoder and decoder switch using someother technique and/or on some other basis.

If the mode is not raw mode, the one bit INVERT field is sent. Inseveral coding modes where conditional inversion may be performed, theINVERT field indicates whether the bits in the bit plane are to beinverted before encoding takes place in the encoder and whether theoutput of decoding in the decoder is to be inverted. The INVERT field is1 when most of the bits in the bit plane are equal to 1, and 0 when mostof the bits in the bit plane are equal to 0. The encoder employs severalcoding modes (such as normal-2 and normal-6) that consume less bits whenmore 0s are present. If the bit plane to be encoded has more 1s than 0s,the encoder can invert the bit plane to increase the proportion of 0s inthe bit plane and increase the potential for bit savings. Other modes(such as diff-2 and diff-6) use the value of the INVERT to calculate apredictor bit plane. Therefore, in some coding modes, the final bitplane reconstructed at the decoder depends on INVERT.

The DATABITS field is an entropy coded stream of VLC symbols containingthe information necessary to reconstruct the bit plane, given the MODEand INVERT fields.

C. Coding Modes

In the second implementation, the encoder encodes binary information(e.g., skipped macroblock information) in any of seven different codingmodes: row-skip mode, column-skip mode, normal-2 mode, normal-6 mode,diff-2 mode, diff-6 mode, and raw mode. A decoder performs correspondingdecoding for any of the seven coding modes. Each mode is described indetail below.

Alternatively, the encoder and decoder use other and/or additionalcoding modes.

1. Row-Skip and Column-Skip Modes

The row-skip coding mode saves bits by representing a row in a bit planewith a single bit if each binary symbol in the row is of a certainvalue. For example, the encoder represents a skipped macroblock with a 0in a bit plane, and uses a row-skip coding mode that represents a row ofall 0s with a single bit. The encoder therefore saves bits when entirerows of macroblocks are skipped. The decoder performs correspondingdecoding.

In the second implementation, all-zero rows are indicated using one bitset to 0. When the row is not all zero, the one bit indicator is set to1, and this is followed by colMB bits containing the bit plane row inorder. Rows are scanned in the natural order.

Likewise, for the column-skip mode, if the entire row is zero, a 0 bitis sent. Else, a 1 is sent, followed by the rowMB bits containing theentire column, in order. Columns are scanned in the natural order.

For coding of leftover rows and/or columns in diff-6 and normal-6 modes(described below), the same logic is applied. A one-bit flag indicateswhether the row or column is all zero. If not, the entire row or columnis transmitted using one bit per symbol.

When the encoder encodes a bit plane consisting primarily of 1s,row-skip and column-skip coding are usually less efficient, because ofthe lower probability that rows/columns will consist entirely of 0s.However, the encoder can perform an inversion on the bit plane in such asituation to increase the proportion of 0s and potentially increase bitsavings. Thus, when conditional inversion is indicated through theINVERT bit, the encoder pre-inverts the bit plane before the bit planeis tiled and coded. On the decoder side, conditional inversion isimplemented by taking the inverse of the final output. (This is notperformed for the diff-2 and diff-6 mode.)

FIG. 19 shows a technique (1900) for encoding binary information in abit plane in a row-skip coding mode. The encoder first checks whetherinversion of the bit plane is appropriate, and, if so, performs theinversion (1910). The encoder then checks a row in the bit plane to seeif each bit in the row is equal to 0 (1920). If so, the encoder sets theindicator bit for the row to 0 (1930). If any of the bits in the row arenot 0, the encoder sets the indicator bit for the row to 1 and encodeseach bit in the row with one bit (1940). When the encoder is doneencoding all rows in the bit plane (1950), encoding of the bit planeends.

A decoder performs corresponding decoding for the row-skip coding mode.

FIG. 20 shows a technique for encoding binary information in acolumn-skip coding mode. The encoder first checks whether inversion ofthe bit plane is appropriate, and, if so, performs the inversion (2010).The encoder then checks a column in the bit plane to see if each bit inthe column is equal to 0 (2020). If so, the encoder sets the indicatorbit for the column to 0 (2030). If any of the bits in the column are not0, the encoder sets the indicator bit for the column to 1 and encodeseach bit in the column with one bit (1940). When the encoder is doneencoding all columns in the bit plane (1950), encoding of the bit planeends.

A decoder performs corresponding decoding for the column-skip codingmode.

2. Normal-2 Mode

The encoder uses the normal-2 mode to jointly encode plural binarysymbols in a bit plane (e.g., by using a vector Huffman or othervariable length encoding scheme). The encoder encodes pairs of binarysymbols with variable length codes. The decoder performs correspondingdecoding.

If rowMB×colMB is odd, the first symbol is encoded as a single bit.Subsequent symbols are encoded pairwise, in natural scan order. A VLCtable is used to encode the symbol pairs to reduce overall entropy.

When conditional inversion is indicated through the INVERT bit, theencoder pre-inverts the bit plane before the bit plane is codedpairwise. On the decoder side, conditional inversion is implemented bytaking the inverse of the final output. (When the diff-2 mode is used,conditional inversion is not performed at this step.)

FIG. 21 shows a technique (2100) for encoding binary information innormal-2 mode. The encoder performs an initial check to determinewhether inversion of the bit plane is appropriate to improve codingefficiency and, if so, performs the inversion (2110). The encoder thendetermines whether the bit plane being coded has an odd number of binarysymbols (2120). If so, the encoder encodes the first symbol with asingle bit (2130). The encoder then encodes symbol pairs with variablelength codes, using shorter codes to represent more probable pairs andlonger codes to represent less probable pairs (2140). When encoding ofthe symbol pairs is done (2150), the encoding ends.

A decoder performs corresponding decoding for the normal-2 coding mode.

3. Normal-6 Mode

The encoder also uses the normal-6 mode to jointly encode plural binarysymbols in a bit plane (e.g., by using a vector Huffman or othervariable length encoding scheme). The encoder tiles groups of six binarysymbols and represents each group with a variable length code. Thedecoder performs corresponding decoding.

In the normal-6 mode (and the diff-6 mode), the bit plane is encoded ingroups of six pixels. These pixels are grouped into either 2×3 or 3×2tiles. The bit plane is tiled maximally using a set of rules, and theremaining pixels are encoded using variants of row-skip and column-skipmodes.

In the second implementation, 3×2 “vertical” tiles are used if and onlyif rowMB is a multiple of 3 and colMB is not a multiple of 3. Otherwise,2×3 “horizontal” tiles are used. FIGS. 22, 23 and 24 show examples offrames tiled in the normal-6 coding mode. FIG. 22 shows a frame (2200)with 3×2 vertical tiles and a 1-symbol wide remainder (shown as a shadedarea) to be coded in column-skip mode. FIG. 23 shows a frame (2300) with2×3 horizontal tiles and a 1-symbol wide remainder to be coded inrow-skip mode. FIG. 24 shows a frame (2400) with 2×3 horizontal tilesand 1-symbol wide remainders to be coded in row-skip and column-skipmodes.

While 3×2 and 2×3 tiles are used in this example, in other embodiments,different configurations of tiles and/or different tiling rules areused.

The 6-element tiles are encoded first, followed by the column-skip androw-skip encoded linear tiles. If the array size is a multiple of 3×2 or2×3, the latter linear tiles do not exist and the bit plane is perfectlytiled. The 6-element rectangular tiles are encoded using a VLC table.

When conditional inversion is indicated through the INVERT bit, theencoder pre-inverts the bit plane before the bit plane is tiled andcoded. On the decoder side, conditional inversion is implemented bytaking the inverse of the final output. (When the diff-6 mode is used,conditional inversion is not performed at this step.)

FIG. 25 shows a technique (2500) for encoding binary information innormal-6 mode. The encoder performs an initial check to determinewhether inversion of the bit plane is appropriate to improve codingefficiency and, if so, performs the inversion (2510). The encoder thenchecks whether the number of rows in the bit plane is a multiple ofthree (2520). If the number of rows is not a multiple of three, theencoder groups the symbols in the bit plane into 2×3 horizontal tiles(2530).

If the number of rows is a multiple of three, the encoder checks whetherthe number of columns in the bit plane is a multiple of three (2540). Ifthe number of columns is a multiple of three, the encoder groups thesymbols in the bit plane into 2×3 horizontal tiles (2530). If the numberof columns is not a multiple of three, the encoder groups the symbolsinto 3×2 vertical tiles (2550).

After grouping symbols into 3×2 or 2×3 tiles, the encoder encodes thegroups of six tiled symbols using a technique such as a six-dimensionvector Huffman coding technique or some other coding technique (2560).The encoder encodes any remaining untiled symbols using the row-skipand/or column-skip coding techniques described above (2570).

A decoder performs corresponding decoding for the normal-6 coding mode.

In other embodiments, an encoder uses other techniques to code the tiledand untiled symbols.

4. Diff-2 and Diff-6 Modes

Differential coding modes such as diff-2 or diff-6 mode encode bitplanes by first generating a bit plane of differential (or residual)bits for the bit plane to be coded, based on a predictor for the bitplane to be coded. The residual bit plane is then encoded using, forexample, the normal-2 or normal-6 coding mode, without conditionalinversion.

In the second implementation, the diff-2 and diff-2 modes employdifferential coding denoted by the operation diff. If eitherdifferential mode is used, a bit plane of differential bits is firstgenerated by examining the predictor {circumflex over (b)}(i, j) of thebit plane b(i, j), which is defined as the causal operation:

$\begin{matrix}{{\hat{b}\left( {i,j} \right)} = \left\{ {\begin{matrix}{INVERT} & {{i = {j = 0}},{{{or}\mspace{14mu}{b\left( {i,{j - 1}} \right)}} \neq {b\left( {{i - 1},j} \right)}}} \\{b\left( {0,{j - 1}} \right)} & {i = 0} \\{b\left( {{i - 1},j} \right)} & {otherwise}\end{matrix}.} \right.} & (1)\end{matrix}$

In other words, the predictor {circumflex over (b)}(i, j) of a givenbinary symbol b(i, j) will be the binary symbol just to the left b(i−1,j) except in the following special cases:

-   -   1) If b(i, j) is at the top left corner of the bit plane, or if        the above binary symbol b(i, j−1) is not equal to the binary        symbol to the left b(i−1, j), the predictor {circumflex over        (b)}(i, j) is equal to the value of INVERT; or    -   2) If 1) does not apply and b(i, j) is in the left column        (i==0), the predictor {circumflex over (b)}(i, j) will be the        above binary symbol b(i, j−1).

On the encoder side, the diff operation computes the residual bit planer according to:r(i,j)=b(i,j)⊕{circumflex over (b)}(i,j)  (2),where ⊕ is the exclusive or operation. The residual bit plane is encodedusing the normal-2 or normal-6 modes with no conditional inversion.

On the decoder side, the residual bit plane is regenerated using theappropriate normal mode. Subsequently, the residual bits are used toregenerate the original bit plane as the binary 2-D difference:b(i,j)=r(i,j)⊕{circumflex over (b)}(i,j)  (3).

FIG. 26 shows a technique (2600) for encoding binary information in adifferential coding mode. The encoder calculates a predictor for a bitplane (2610), for example, as shown in equation 1. The encoder thencalculates a residual bit plane, for example, by performing an XORoperation on the bit plane and its predictor (2620). The encoder thenencodes the residual bit plane (e.g., in normal-2 or normal-6 mode)(2630).

FIG. 27 shows a technique (2700) for decoding binary information encodedin a differential coding mode. The decoder decodes the residual bitplane (2710) using an appropriate decoding technique, based on the modeused to encode the residual bit plane (e.g., normal-2 or normal-6 mode).The decoder also calculates the predictor for the bit plane (2720),using the same technique used in the encoder. The decoder thenreconstructs the original bit plane, for example, by performing an XORoperation on the decoded residual bit plane and the predictor bit plane(2730).

5. Raw Mode

All modes except for raw mode encode a bit plane at the frame level,which calls for a second pass through the frame during encoding.However, for low-latency situations, the second pass can addunacceptable delay (e.g., because transmission of the frame header andmacroblock layer information is delayed until the last macroblock in theframe is reached, because of the time spent encoding the bit plane).

The raw mode uses the traditional method of encoding the bit plane onebit per binary symbol at the same location in the bitstream as the restof the macroblock level information. Although macroblock-level coding ofsymbols is not a new concept by itself, switching the coding of symbolsfrom frame level to macroblock level provides a low latency alternativeto frame-level coding.

FIG. 28 shows a technique (2800) for selectively encoding binaryinformation for a macroblock in raw coding mode for low latencyapplications. First, the encoder checks whether to use raw mode toencode the binary information (2810). If so, the encoder encodes a bitat the macroblock level for a macroblock (2820) and checks whether themacroblock is the last macroblock in the frame (2830). If the macroblockis not the last macroblock in the frame, the encoder continues byencoding a bit for the next macroblock at the macroblock level (2820).

If the encoder does not use the raw coding mode, the encoder encodes abit plane at the frame level for the macroblocks in the frame (2840).When the encoding of the macroblocks in the frame is done (2850), theencoding ends for the frame.

While the technique (2800) shows switching modes on a frame-by-framebasis, alternatively the encoder switches on some other basis.

Having described and illustrated the principles of our invention withreference to various embodiments, it will be recognized that the variousembodiments can be modified in arrangement and detail without departingfrom such principles. It should be understood that the programs,processes, or methods described herein are not related or limited to anyparticular type of computing environment, unless indicated otherwise.Various types of general purpose or specialized computing environmentsmay be used with or perform operations in accordance with the teachingsdescribed herein. Elements of embodiments shown in software may beimplemented in hardware and vice versa.

In view of the many possible embodiments to which the principles of ourinvention may be applied, we claim as our invention all such embodimentsas may come within the scope and spirit of the following claims andequivalents thereto.

We claim:
 1. One or more computer-readable media storingcomputer-executable instructions for causing a computing system, whenprogrammed thereby, to perform operations, wherein the one or morecomputer-readable media are selected from the group consisting ofvolatile memory, non-volatile memory, magnetic disk, CD-ROM, and DVD,the operations comprising: encoding plural video pictures of a videosequence to produce encoded data, the plural video pictures includingplural predicted macroblocks, wherein each block of the plural predictedmacroblocks is predicted from no more than one reference video picture,including: selecting a coding mode from among plural available codingmodes; processing one or more skipped macroblocks among the pluralpredicted macroblocks, wherein each of the one or more skippedmacroblocks uses causally predicted motion for the skipped macroblockbased upon motion of one or more other predicted macroblocks around theskipped macroblock, and wherein each of the one or more skippedmacroblocks lacks residual information; and encoding skipped macroblockinformation for the one or more skipped macroblocks for signaling at alayer in bitstream syntax, wherein the skipped macroblock informationindicates skipped/not skipped status, and wherein the encoding includesencoding the skipped macroblock information according to the coding modeselected from among the plural available coding modes; and outputtingthe encoded data in a bitstream.
 2. The one or more computer-readablemedia of claim 1, wherein the processing the one or more skippedmacroblocks includes, for a current macroblock of a current P-picture ofthe plural video pictures, predicting the current macroblock using amotion vector equal to a predictor motion vector derived from motionvectors of the one or more other predicted macroblocks, which neighborthe current macroblock in the current P-picture.
 3. The one or morecomputer-readable media of claim 1, wherein a current P-picture of theplural video pictures includes the one or more skipped macroblocks andthe one or more other predicted macroblocks, and wherein the processingthe one or more skipped macroblocks comprises, for each of the one ormore skipped macroblocks: determining that the causally predicted motionfor the skipped macroblock is equal to a predictor motion vector derivedfrom motion vectors of the one or more other predicted macroblocksaround the skipped macroblock in the current P-picture; and determiningthat the skipped macroblock lacks residual information.
 4. The one ormore computer-readable media of claim 1, wherein the bitstream syntaxincludes picture layer and macroblock layer.
 5. The one or morecomputer-readable media of claim 4, wherein the layer at which theskipped macroblock information is signaled is the picture layer.
 6. Theone or more computer-readable media of claim 5, wherein a header at thepicture layer includes the skipped macroblock information.
 7. The one ormore computer-readable media of claim 4, wherein the layer at which theskipped macroblock information is signaled is below the picture layer.8. The one or more computer-readable media of claim 1, wherein theplural available coding modes include a row-skip coding mode, acolumn-skip coding mode, a raw coding mode, a symbol pair coding mode,and a symbol group coding mode.
 9. One or more computer-readable mediastoring computer-executable instructions for causing a computing system,when programmed thereby, to perform operations, wherein the one or morecomputer-readable media are selected from the group consisting ofvolatile memory, non-volatile memory, magnetic disk, CD-ROM, and DVD,the operations comprising: receiving encoded data in a bit stream forplural video pictures of a video sequence, the plural video picturesincluding plural predicted macroblocks, wherein each block of the pluralpredicted macroblocks is predicted from no more than one reference videopicture; and decoding the plural video pictures, including: selecting acoding mode from among plural available coding modes; for one or moreskipped macroblocks among the plural predicted macroblocks, decodingskipped macroblock information signaled at a layer in bitstream syntax,the skipped macroblock information having been encoded according to thecoding mode selected from among the plural available coding modes,wherein the skipped macroblock information indicates skipped/not skippedstatus; and processing the one or more skipped macroblocks, wherein eachof the one or more skipped macroblocks uses causally predicted motionfor the skipped macroblock based upon motion of one or more otherpredicted macroblocks around the skipped macroblock, and wherein each ofthe one or more skipped macroblocks lacks residual information.
 10. Theone or more computer-readable media of claim 9, wherein the processingthe one or more skipped macroblocks includes, for a current macroblockof a current P-picture of the plural video pictures, reconstructing thecurrent macroblock using a motion vector equal to a predictor motionvector derived from motion vectors of the one or more other predictedmacroblocks, which neighbor the current macroblock in the currentP-picture.
 11. The one or more computer-readable media of claim 9,wherein the bitstream syntax includes picture layer and macroblocklayer.
 12. The one or more computer-readable media of claim 11, whereinthe layer at which the skipped macroblock information is signaled is thepicture layer.
 13. The one or more computer-readable media of claim 12,wherein a header at the picture layer includes the skipped macroblockinformation.
 14. The one or more computer-readable media of claim 11,wherein the layer at which the skipped macroblock information issignaled is below the picture layer.
 15. The one or morecomputer-readable media of claim 9, wherein the plural available codingmodes include a row-skip coding mode, a column-skip coding mode, a rawcoding mode, a symbol pair coding mode, and a symbol group coding mode.16. A computer-readable medium having stored thereon encoded data in abitstream for plural video pictures of a video sequence, the pluralvideo pictures including plural predicted macroblocks, wherein eachblock of the plural predicted macroblocks is predicted from no more thanone reference video picture, and wherein the computer-readable medium isvolatile memory, non-volatile memory, a magnetic disk, a CD-ROM, or aDVD, the encoded data being organized for decoding operations thatinclude selecting a coding mode from among plural available codingmodes; for one or more skipped macroblocks among the plural predictedmacroblocks, decoding skipped macroblock information signaled at a layerin bitstream syntax, the skipped macroblock information having beenencoded according to the coding mode selected from among the pluralavailable coding modes, wherein the skipped macroblock informationindicates skipped/not skipped status; and processing the one or moreskipped macroblocks, wherein each of the one or more skipped macroblocksuses causally predicted motion for the skipped macroblock based uponmotion of one or more other predicted macroblocks around the skippedmacroblock, and wherein each of the one or more skipped macroblockslacks residual information.
 17. The computer-readable medium of claim16, wherein the processing the one or more skipped macroblocks includes,for a current macroblock of a current P-picture of the plural videopictures, reconstructing the current macroblock using a motion vectorequal to a predictor motion vector derived from motion vectors of theone or more other predicted macroblocks, which neighbor the currentmacroblock in the current P-picture.
 18. The computer-readable medium ofclaim 16, wherein the bitstream syntax includes picture layer andmacroblock layer, and wherein the layer at which the skipped macroblockinformation is signaled is the picture layer.
 19. The computer-readablemedium of claim 16, wherein the bitstream syntax includes picture layerand macroblock layer, and wherein the layer at which the skippedmacroblock information is signaled is below the picture layer.
 20. Thecomputer-readable medium of claim 16, wherein the plural availablecoding modes include a row-skip coding mode, a column-skip coding mode,a raw coding mode, a symbol pair coding mode, and a symbol group codingmode.